<!DOCTYPE html>
<html lang="" xml:lang="">
  <head>
    <title>Inference overview</title>
    <meta charset="utf-8" />
    <meta name="author" content="datasciencebox.org" />
    <script src="libs/header-attrs/header-attrs.js"></script>
    <link href="libs/font-awesome/css/all.css" rel="stylesheet" />
    <link href="libs/font-awesome/css/v4-shims.css" rel="stylesheet" />
    <link href="libs/panelset/panelset.css" rel="stylesheet" />
    <script src="libs/panelset/panelset.js"></script>
    <link rel="stylesheet" href="../xaringan-themer.css" type="text/css" />
    <link rel="stylesheet" href="../slides.css" type="text/css" />
  </head>
  <body>
    <textarea id="source">
class: center, middle, inverse, title-slide

.title[
# Inference overview
]
.subtitle[
## <br><br> Data Science in a Box
]
.author[
### <a href="https://datasciencebox.org/">datasciencebox.org</a>
]

---





layout: true
  
&lt;div class="my-footer"&gt;
&lt;span&gt;
&lt;a href="https://datasciencebox.org" target="_blank"&gt;datasciencebox.org&lt;/a&gt;
&lt;/span&gt;
&lt;/div&gt; 

---



class: middle

# Inference overview

---

## What do you want to do?

- Estimation -&gt; Confidence interval
- Decision -&gt; Hypothesis test
- First step: Ask the following questions
  - How many variables?
  - What types of variables?
  - What is the research question?

---

class: middle

# Confidence intervals

---

## Confidence intervals

- Bootstrap
- Bounds: cutoff values for the middle XX% of the distribution
- Interpretation: We are XX% confident that the true population parameter is in the interval.
- Definition of confidence level: XX% of random samples of size n are expected to produce confidence intervals that contain the true population parameter.
- `infer::generate(reps, type = "bootstrap")`

---

## Confidence intervals exercises

.question[
Describe the simulation process for estimating the parameter assigned to your team.

- Note any assumptions you make in terms of sample size, observed sample statistic, etc.
- Imagine using index cards or chips to represent the data. 
- Specify whether the simulation type would be bootstrap, simulate, or permute.

]

---

## Accuracy vs. precision

.question[
What happens to the width of the confidence interval as the confidence level increases? Why? Should we always prefer a confidence interval with a higher confidence level?
]

---

## Sample size and width of intervals

&lt;img src="u4-d13-inference-overview_files/figure-html/unnamed-chunk-2-1.png" width="60%" style="display: block; margin: auto;" /&gt;

---

## Sample size and width of intervals

&lt;img src="u4-d13-inference-overview_files/figure-html/unnamed-chunk-3-1.png" width="60%" style="display: block; margin: auto;" /&gt;

---

## Sample size and width of intervals

&lt;img src="u4-d13-inference-overview_files/figure-html/unnamed-chunk-4-1.png" width="60%" style="display: block; margin: auto;" /&gt;

---

## Equivalency of confidence and significance levels

- Two sided alternative HT with `\(\alpha\)` `\(\rightarrow\)` `\(CL = 1 - \alpha\)`
- One sided alternative HT with `\(\alpha\)` `\(\rightarrow\)` `\(CL = 1 - (2 \times \alpha)\)`

&lt;img src="u4-d13-inference-overview_files/figure-html/unnamed-chunk-5-1.png" width="60%" style="display: block; margin: auto;" /&gt;

---

## Interpretation of confidence intervals

.question[
Which of the following is more informative: 

&lt;ul&gt;
&lt;li&gt; The difference in price of a gallon of milk between Whole Foods and Harris Teeter is 30 cents.
&lt;li&gt; A gallon of milk costs 30 cents more at Whole Foods compared to Harris Teeter.
&lt;/ul&gt;
&lt;/div&gt;
]

--

.question[
What does your answer tell you about interpretation of confidence intervals for differences between two population parameters?
]

---

class: middle

# Hypothesis tests

---

## Hypothesis testing

- Set the hypotheses.
- Calculate the observed sample statistic.
- Calculate the p-value.
- Make a conclusion, about the hypotheses, in context of the data and the research question.
- `infer::hypothesize(null = "point")` and `infer::generate(reps, type = "simulate")` or `infer::generate(reps, type = "bootstrap")`
- `infer::hypothesize(null = "independence")` and `infer::generate(reps, type = "permute")`
  
---

## Hypothesis testing exercises

.question[
Describe the simulation process for tesing for the parameter assigned to your team.

- Note any assumptions you make in terms of sample size, observed sample statistic, etc.
- Imagine using index cards or chips to represent the data. 
- Specify whether the null hypothesis would be independence or point.
- Specify whether the simulation type would be bootstrap, simulate, or permute.
]

---

## Testing errors

- Type 1: Reject `\(H_0\)` when you shouldn't have
    - P(Type 1 error) = `\(\alpha\)`
- Type 2: Fail to reject `\(H_0\)` when you should have
    - P(Type 2 error) is harder to calculate, but increases as `\(\alpha\)` decreases

--

.question[
In a court of law

- Null hypothesis: Defendant is innocent
- Alternative hypothesis: Defendant is guilty

Which is worse: Type 1 or Type 2 error?
]

---

## Probabilities of testing errors

- P(Type 1 error) = `\(\alpha\)`
- P(Type 2 error) = 1 - Power
- Power = P(correctly rejecting the null hypothesis)

--

.question[
Fill in the blanks in terms of correctly/incorrectly rejecting/failing to reject the null hypothesis:

- `\(\alpha\)` is the probability of ______.
- 1 - Power is the probability of ______.
]

---

class: middle

# Examples

---

## What do you want to do?

- Estimation -&gt; Confidence interval
- Decision -&gt; Hypothesis test
- First step: Ask the following questions
  - How many variables?
  - What type(s) of variable(s)?
  - What is the research question?

---

## Data: NC births

The dataset is in the `openintro` package.


```r
glimpse(ncbirths)
```

```
## Rows: 1,000
## Columns: 13
## $ fage           &lt;int&gt; NA, NA, 19, 21, NA, NA, 18, 17, NA, 20, …
## $ mage           &lt;int&gt; 13, 14, 15, 15, 15, 15, 15, 15, 16, 16, …
## $ mature         &lt;fct&gt; younger mom, younger mom, younger mom, y…
## $ weeks          &lt;int&gt; 39, 42, 37, 41, 39, 38, 37, 35, 38, 37, …
## $ premie         &lt;fct&gt; full term, full term, full term, full te…
## $ visits         &lt;int&gt; 10, 15, 11, 6, 9, 19, 12, 5, 9, 13, 9, 8…
## $ marital        &lt;fct&gt; not married, not married, not married, n…
## $ gained         &lt;int&gt; 38, 20, 38, 34, 27, 22, 76, 15, NA, 52, …
## $ weight         &lt;dbl&gt; 7.63, 7.88, 6.63, 8.00, 6.38, 5.38, 8.44…
## $ lowbirthweight &lt;fct&gt; not low, not low, not low, not low, not …
## $ gender         &lt;fct&gt; male, male, female, male, female, male, …
## $ habit          &lt;fct&gt; nonsmoker, nonsmoker, nonsmoker, nonsmok…
## $ whitemom       &lt;fct&gt; not white, not white, white, white, not …
```

---

## Length of gestation

&lt;img src="u4-d13-inference-overview_files/figure-html/unnamed-chunk-7-1.png" width="60%" style="display: block; margin: auto;" /&gt;


```
## # A tibble: 1 × 7
##     min  xbar   med     s    q1    q3   max
##   &lt;int&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;int&gt;
## 1    20  38.3    39  2.93    37    40    45
```


---

## Length of gestation

.question[
Assuming that this sample is representative of all births in NC, we are 95% confident that the average length of gestation for babies in NC is between ---- and ---- weeks.
]

--

**(1) How many variables?**

--

1 variable: length of gestation, `weeks`

--

**(2) What type(s) of variable(s)?**

--

Numerical

--

**(3) What is the research question?**

--

Estimate the average length of gestation `\(\rightarrow\)` confidence interval

---

## Simulation for CI for a mean

**Goal:** Use bootstrapping to estimate the sampling variability of the mean, i.e. the variability of means taken from the same population with the same sample size.

--

1. Take a bootstrap sample - a random sample taken with replacement from the 
original sample, of the same size as the original sample.

2. Calculate the mean of the bootstrap sample.

3. Repeat steps (1) and (2) many times to create a bootstrap distribution - 
a distribution of bootstrap means.

4. Calculate the bounds of the 95% confidence interval as the middle 95% 
of the bootstrap distribution.

---

## Set a seed first

From the documentation of `set.seed`:

- `set.seed` uses a single integer argument to set as many seeds as are required. There is no guarantee that different values of seed will seed the RNG differently, although any exceptions would be extremely rare.
- Initially, there is no seed; a new one is created from the current time and the process ID when one is required. Hence different sessions will give different simulation results, by default.


```r
set.seed(20180326)
```

---

## Computation for CI for a mean


```r
boot_means &lt;- ncbirths %&gt;%
  filter(!is.na(weeks)) %&gt;% # remove NAs
  specify(response = weeks) %&gt;%
  generate(reps = 1000, type = "bootstrap") %&gt;%
  calculate(stat = "mean")
ggplot(data = boot_means, aes(x = stat)) +
  geom_histogram(binwidth = 0.03)
```

&lt;img src="u4-d13-inference-overview_files/figure-html/unnamed-chunk-10-1.png" width="60%" style="display: block; margin: auto;" /&gt;

---

## Length of gestation


```r
boot_means %&gt;%
  summarise(
    lower = quantile(stat, 0.025),
    upper = quantile(stat, 0.975)
  )
```

```
## # A tibble: 1 × 2
##   lower upper
##   &lt;dbl&gt; &lt;dbl&gt;
## 1  38.2  38.5
```

--

Assuming that this sample is representative of all births in NC, we are 95% confident that the average length of gestation for babies in NC is between 38.1 and 38.5 weeks.

---

## Length of gestation, revisited

.question[
The average length of human gestation is 280 days, or 40 weeks, from the first day of the woman's last menstrual period. Do these data provide convincing evidence that average length of gestation for women in NC is different than 40 weeks? Use a significance level of 5%.
]

--

`\(H_0: \mu = 40\)`  
`\(H_A: \mu \ne 40\)`

--

- We just said, "we are 95% confident that the average length of gestation for babies in NC is between 38.1 and 38.5 weeks".

- Since the null value is outside the CI, we would reject the null hypothesis in favor of the alternative.

- But an alternative, more direct, way of answering this question is using a hypothesis test.

---

## Simulation for HT for a mean

**Goal:** Use bootstrapping to generate a sampling distribution under the assumption of the null hypothesis being true. Then, calculate the p-value to make a decision on the hypotheses.

--

1. Take a bootstrap sample - a random sample taken with replacement from the 
original sample, of the same size as the original sample.

2. Calculate the mean of the bootstrap sample.

3. Repeat steps (1) and (2) many times to create a bootstrap distribution - 
a distribution of bootstrap means.

4. Shift the bootstrap distribution to be centered at the null value by subtracting/adding the difference between the center of the bootstrap distribution and the null value to each bootstrap mean.

5. Calculate the p-value as the proportion of simulations that yield a sample mean at least as extreme as the observed sample mean.

---

## Computation for HT for a mean


```r
boot_means_shifted &lt;- ncbirths %&gt;%
  filter(!is.na(weeks)) %&gt;% # remove NAs
  specify(response = weeks) %&gt;%
  hypothesize(null = "point", mu = 40) %&gt;% # hypothesize step
  generate(reps = 1000, type = "bootstrap") %&gt;%
  calculate(stat = "mean")
ggplot(data = boot_means_shifted, aes(x = stat)) +
  geom_histogram(binwidth = 0.03) +
  geom_vline(xintercept = 38.33, color = "red") +
  geom_vline(xintercept = 40 + (40 - 38.33), color = "red")
```

&lt;img src="u4-d13-inference-overview_files/figure-html/unnamed-chunk-12-1.png" width="50%" style="display: block; margin: auto;" /&gt;

---

## Length of gestation


```r
boot_means_shifted %&gt;%
  filter(stat &lt;= 38.33) %&gt;%
  summarise(p_value = 2 * (n() / 1000))
```

```
## # A tibble: 1 × 1
##   p_value
##     &lt;dbl&gt;
## 1       0
```

--

Since p-value less than the significance level, we reject the null hypothesis. The data provide convincing evidence that the average length of gestation of births in NC is different than 40.

---

## **infer** structure


```r
df %&gt;%
  specify(response, explanatory) %&gt;% # explanatory optional
  generate(reps, type) %&gt;% # type: bootstrap, simulate, or permute
  calculate(stat)
```

- Always start with data frame
- Result is always a data frame with a variable called `stat`
   - See the documentation for `calculate` to see which `stat`istics can be calculated
- For hypothesis testing add a `hypothesize()` step between `specify()` and `generate()`
    - `null = "point"`, and then specify the null value
    - `null = "independence"`
    </textarea>
<style data-target="print-only">@media screen {.remark-slide-container{display:block;}.remark-slide-scaler{box-shadow:none;}}</style>
<script src="https://remarkjs.com/downloads/remark-latest.min.js"></script>
<script>var slideshow = remark.create({
"ratio": "16:9",
"highlightLines": true,
"highlightStyle": "solarized-light",
"countIncrementalSlides": false
});
if (window.HTMLWidgets) slideshow.on('afterShowSlide', function (slide) {
  window.dispatchEvent(new Event('resize'));
});
(function(d) {
  var s = d.createElement("style"), r = d.querySelector(".remark-slide-scaler");
  if (!r) return;
  s.type = "text/css"; s.innerHTML = "@page {size: " + r.style.width + " " + r.style.height +"; }";
  d.head.appendChild(s);
})(document);

(function(d) {
  var el = d.getElementsByClassName("remark-slides-area");
  if (!el) return;
  var slide, slides = slideshow.getSlides(), els = el[0].children;
  for (var i = 1; i < slides.length; i++) {
    slide = slides[i];
    if (slide.properties.continued === "true" || slide.properties.count === "false") {
      els[i - 1].className += ' has-continuation';
    }
  }
  var s = d.createElement("style");
  s.type = "text/css"; s.innerHTML = "@media print { .has-continuation { display: none; } }";
  d.head.appendChild(s);
})(document);
// delete the temporary CSS (for displaying all slides initially) when the user
// starts to view slides
(function() {
  var deleted = false;
  slideshow.on('beforeShowSlide', function(slide) {
    if (deleted) return;
    var sheets = document.styleSheets, node;
    for (var i = 0; i < sheets.length; i++) {
      node = sheets[i].ownerNode;
      if (node.dataset["target"] !== "print-only") continue;
      node.parentNode.removeChild(node);
    }
    deleted = true;
  });
})();
// add `data-at-shortcutkeys` attribute to <body> to resolve conflicts with JAWS
// screen reader (see PR #262)
(function(d) {
  let res = {};
  d.querySelectorAll('.remark-help-content table tr').forEach(tr => {
    const t = tr.querySelector('td:nth-child(2)').innerText;
    tr.querySelectorAll('td:first-child .key').forEach(key => {
      const k = key.innerText;
      if (/^[a-z]$/.test(k)) res[k] = t;  // must be a single letter (key)
    });
  });
  d.body.setAttribute('data-at-shortcutkeys', JSON.stringify(res));
})(document);
(function() {
  "use strict"
  // Replace <script> tags in slides area to make them executable
  var scripts = document.querySelectorAll(
    '.remark-slides-area .remark-slide-container script'
  );
  if (!scripts.length) return;
  for (var i = 0; i < scripts.length; i++) {
    var s = document.createElement('script');
    var code = document.createTextNode(scripts[i].textContent);
    s.appendChild(code);
    var scriptAttrs = scripts[i].attributes;
    for (var j = 0; j < scriptAttrs.length; j++) {
      s.setAttribute(scriptAttrs[j].name, scriptAttrs[j].value);
    }
    scripts[i].parentElement.replaceChild(s, scripts[i]);
  }
})();
(function() {
  var links = document.getElementsByTagName('a');
  for (var i = 0; i < links.length; i++) {
    if (/^(https?:)?\/\//.test(links[i].getAttribute('href'))) {
      links[i].target = '_blank';
    }
  }
})();
// adds .remark-code-has-line-highlighted class to <pre> parent elements
// of code chunks containing highlighted lines with class .remark-code-line-highlighted
(function(d) {
  const hlines = d.querySelectorAll('.remark-code-line-highlighted');
  const preParents = [];
  const findPreParent = function(line, p = 0) {
    if (p > 1) return null; // traverse up no further than grandparent
    const el = line.parentElement;
    return el.tagName === "PRE" ? el : findPreParent(el, ++p);
  };

  for (let line of hlines) {
    let pre = findPreParent(line);
    if (pre && !preParents.includes(pre)) preParents.push(pre);
  }
  preParents.forEach(p => p.classList.add("remark-code-has-line-highlighted"));
})(document);</script>

<script>
slideshow._releaseMath = function(el) {
  var i, text, code, codes = el.getElementsByTagName('code');
  for (i = 0; i < codes.length;) {
    code = codes[i];
    if (code.parentNode.tagName !== 'PRE' && code.childElementCount === 0) {
      text = code.textContent;
      if (/^\\\((.|\s)+\\\)$/.test(text) || /^\\\[(.|\s)+\\\]$/.test(text) ||
          /^\$\$(.|\s)+\$\$$/.test(text) ||
          /^\\begin\{([^}]+)\}(.|\s)+\\end\{[^}]+\}$/.test(text)) {
        code.outerHTML = code.innerHTML;  // remove <code></code>
        continue;
      }
    }
    i++;
  }
};
slideshow._releaseMath(document);
</script>
<!-- dynamically load mathjax for compatibility with self-contained -->
<script>
(function () {
  var script = document.createElement('script');
  script.type = 'text/javascript';
  script.src  = 'https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-MML-AM_CHTML';
  if (location.protocol !== 'file:' && /^https?:/.test(script.src))
    script.src  = script.src.replace(/^https?:/, '');
  document.getElementsByTagName('head')[0].appendChild(script);
})();
</script>
  </body>
</html>
